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SECTION  1.  BACKGROUND 


TRAC  provided  analysis  support  to  CENTCOM  Forward- Jordon  (CF-J)  in  2014 
to  inform  and  support  analysis  on  Humanitarian  Assistance  (HA)  related  decisions  to 
improve  conditions  in  southern  Syria.  Analysis  was  led  and  conducted  by  Ms.  Sarah 
Holden  at  TRAC-WSMR.  In  July  to  October  2014  TRAC  conducted  a  technical  review 
of  the  products  delivered  to  CF-J. 

Mr.  Leroy  Jackson  led  the  technical  review  and  solicited  anonymous  observations 
from  a  half  dozen  TRAC  analysts  primarily  at  TRAC-MTRY  and  one  NPS  faculty 
member.  This  review  also  served  a  proof  of  principle  for  the  formal  TRAC  Technical 
Review  Program  to  be  implemented  in  FY15. 

1.1.  PROBLEM  STATEMENT 

The  inflow  of  refugees  and  conflict  spill  over  resulting  from  the  Syrian  Civil  War 
is  causing  instability  in  Jordan.  The  US  Embassy-Amman  has  asked  CENTCOM 
Forward- Jordan  (CF-J)  to  inform  and  support  them  on  Humanitarian  Assistance  (HA) 
related  decisions  to  improve  conditions  in  southern  Syria. 

1.2.  OBJECTIVE 

To  enable  more  effective  HA  by  improving  CF-J’s  ability  to  inform  US  Embassy- 
Amman  decisions  about  HA  to  southern  Syria. 

1.3.  SCOPE 

TRAC  will  provide  an  assessment  framework  to  CF-J  that  includes  measures  and 
data  sources  to  identify  the  basic  needs,  match  those  needs  with  appropriate  type  and 
quantity  of  HA,  and  assess  effectiveness  after  HA  has  been  delivered  to  refine 
coordination. 

1.4.  PRODUCTS 

TRAC  delivered  two  sets  of  products  as  listed  below. 
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1.4.1  First  deliverable  to  CF-J. 

•  Data  Collection  Management  Plan  (DCMP  worksheet). 

•  Study  questions,  measures,  and  essential  data  elements  (EDEs). 

•  Potential  data  sources  and  sources  associated  assessment  score. 

•  Data  source  assessment  details. 

•  Data  Source  List  and  Assessment  (DataSrcAssessment  worksheet). 

•  Data  Sources  &  Source  Assessment. 

1.4.2  Second  deliverable  to  CF-J. 

•  Assessment  Framework  (tools)  and  User  Guides. 

•  Forecasting  (CAS,  REF,  REQHA  &  VIOevents). 

•  Simple  Regression  (CASREF,  CASREF_bsoclOFebl4,  REQREF, 
VIOCAS  &  VIOREF). 

•  Multiple  Linear  Regression  (CASQREF,  CASVIOREF,  VIOREQREF). 
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SECTION  2.  TECHNICAL  REVIEW  PROBLEM  STATEMENT 


2.1.  PURPOSE 

Review  the  products  provided  to  CF-J  in  order  to  identify  technical  strengths  and 
weaknesses  in  the  analysis  methodologies  and  supporting  documentation. 

2.2.  APPROACH 

The  technical  review  lead  coordinated  the  review  requirements  with  the  WSMR 
analysts.  Prior  attendance  at  the  Syrian  HA  measurement  space  meeting  reduced  the  need 
for  extensive  background  discovery.  The  WSMR  analyst  provided  the  supporting 
documentation  as  delivered  to  CF-J.  There  were  no  milestones  or  other  deadlines 
mandating  a  quick-turn  review. 

The  lead  analyst  solicited  review  from  a  half  dozen  TRAC  analysts  primarily  at 
TRAC-MTRY.  Analysts  were  selected  based  on  education  and  experience.  The  lead 
analyst  also  solicited  an  infonnal  review  from  an  NPS  faculty  member.  Contributing 
analysts  were  asked  to  focus  on  the  technical  merit  of  the  work,  but  to  also  consider  the 
appropriateness  of  observations  for  the  intended  use  by  deployed  analysts  at  CF-J. 
Analysts  were  allowed  to  submit  anonymous  comments  through  an  internet  capability; 
however,  some  analysts  chose  to  email  comments  to  the  review  lead.  Two  TRAC-MTRY 
analysts  also  assisted  in  reviewing  comments  with  the  review  lead. 
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SECTION  3.  METHODOLOGY 


The  graphic  below  depicts  the  draft  TRAC  technical  review  methodology,  which 
is  described  subsequently  in  this  section.  This  review  prototyped  this  methodology  and 
followed  these  six  steps. 


The  observations  by  the  contributors  were  compiled  in  a  spreadsheet.  The 
fourteen  observations  provided  were  parsed  into  seventeen  observations  since  several 
comments  addressed  multiple  topics.  Some  observations  among  the  seventeen  were 
derived  from  multiple  observations  that  were  consolidated. 


um 


Technical  Review  Methodology 


Technical  Review  Program 


1 .  The  leader(s)  identifying  the  review  requirement  direct  developing  the  review 
request  that  includes  the  task  and  purpose,  scope,  suggested  rules  of  engagement 
(including  sensitivity  of  products  for  review),  and  a  technical  description  of  the  expected 
review  products. 
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2.  The  request  for  technical  review  is  processed  by  the  sponsoring  Center 
operations  and  leadership  to  support  (1)  identifying  a  review  lead  and  team  members,  as 
needed,  (2)  rules  of  engagement  for  the  review,  and  (3)  required  timeline  and  other 
supporting  administrative  instructions.  The  TRAC  Technical  Director  reviews  the  request 
and  supports  TRAC  and  Center  leaders  and  operations  staff  planning. 

3.  The  technical  review  lead  initiates  technical  review  distributing  documentation 
and  guidance  while  organizing  the  effort  of  contributors.  TRAC  Technical  Director 
supports  the  technical  review  lead,  provides  technical  guidance,  and  troubleshoots  issues 
related  to  the  review. 

4.  Contributors  including  members  of  the  technical  review  team  review  the 
documentation  and  develop  AAR-style  comments.  The  technical  review  team  reviews 
comments  submitted  by  contributors  and  requests  clarification  where  necessary.  The 
technical  review  team  consolidates  comments  into  emerging  insights  and  potential 
lessons  identified.  The  technical  review  team  drafts  a  review  report,  typically  a  scripted 
briefing  or  technical  note.  The  Technical  Director  supports  the  review  team  as  necessary. 

5.  The  technical  review  team  briefs  the  technical  review  results  to  responsible 
leader  and  technical  team  representatives.  This  begins  the  process  of  formal  feedback  and 
review  revision  until  the  technical  review  is  approved  and  released  by  the  supporting 
center.  The  technical  review  team  finalizes  the  review  report.  The  Technical  Director 
supports  the  review  team,  center  leaders  and  the  technical  team  representatives. 
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SECTION  4.  REVIEW  RESULTS 


The  review  produced  seventeen  specific  observations.  No  substantial 
weakness  was  found  in  the  technical  products  produced  for  this  project  and  several  areas 
for  sustaining  the  technical  merit  of  the  analysis  were  identified.  This  review  also 
prototyped  the  emerging  TRAC  Technical  Review  Process  to  be  implemented  in  FY15. 

Observations  are  listed  in  the  first  section  below  with  discussion  and,  where 
appropriate,  recommendations.  Recommendations  are  generally  categorized  as  ‘noted’, 
‘improve’  or  ‘sustain’  with  additional  comments  where  appropriate.  Follow-up  questions 
and  answers  for  the  WSMR  analyst  are  listed  in  the  second  section. 

4,1  OBSERVATIONS 

4.1.1  Casualty  Prediction 

Observation.  From  the  report  it  was  not  clear  what  evidence  and  assumptions 
support  the  belief  that  forecasting  casualties  is  valid. 

Discussion.  Substantial  evidence  supports  the  claim  that  violence  and  lack  of  HA 
are  the  main  reasons  why  refugees  come  from  Syria  into  Jordan.  Also,  a  study  using 
violent  events  and  regression  (VIOev)  used  violence  successfully.  It  can  be  reasonably 
assumed  that  the  system  is  in  a  steady  state  so  that  time-series  methods  apply  over  the 
applicable  time  periods. 

Recommendation.  Noted;  consider  an  assumption  about  time  series  analysis  and 
whether  the  system  is  in  a  steady-state  allowing  prediction. 

4.1.2  Weighting  Method 

Observation.  The  weighting  method  is  not  straightforward  and  it  is  unclear  in  an 
initial  reading  how  someone  might  alter  the  weights. 

Discussion.  Weighted  moving  average  requires  that  the  split  be  at  the  midpoint  of 
the  time  period  for  the  recent  and  older  weight.  This  is  an  Excel  limitation  and  the  HA 
team  desired  to  develop  a  tool  in  an  application  that  was  both  familiar  to  the  deploying 
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CF-J  analysts  and  easily  accessible  on  their  computers.  The  effect  is  not  judged  to  be 
relevant  given  other  data  limitations. 

Recommendation.  Noted. 

4.1.3  Phrasing  Assumptions 

Observation.  In  phrasing  assumptions,  consider  phrasing  like,  "Databases  and 
reports  from  government  and  NGOs  are  sufficiently  reliable  and  valid",  rather  than 
"Databases  and  reports  from  government  and  NGOs  are  reliable  and  valid" 

Discussion.  None. 

Recommendation.  Improve  per  observation. 

4.1.4  Problem  Formulation 

Observation.  Excellent  formulation  of  the  final  deliverable  problem  statement, 
objective,  scope  and  assessment  questions. 

Discussion.  None. 

Recommendation.  Sustain. 

4.1.5  Data  Collection  Management  Plan  (DCMP) 

Observation.  DCMP  and  associated  data  source  assessment  and  data  source  score 
breakdown  are  methodologically  sound.  The  level  of  detail  seems  appropriate.  Preference 
for  CF-J  data  is  explained  and  is  justifiable.  Consider  addressing  potential  options  for  use 
of  data  available  from  multiple  source  data  such  as  validation  or  explaining  why  this 
approach  is  not  viable  or  appropriate. 

Discussion.  Lead  WSMR  analyst  agrees  with  the  observation.  Potential  uses 
included  as  background  or  context  or  supporting  validation.  For  validity  the  analyst  did 
develop  categories  that  range  from  ‘self-reported’  to  ‘reported  and  used  by  others’.  The 
HA  team  desired  the  opportunity  to  develop  a  data  testing  and  validation  tool  to  include 
with  this  project;  however  resources  for  the  project  were  reduced  and  only  those 


7 


capabilities  deemed  imperative  were  developed.  Questions  will  remain  about  reliability 
of  the  source  versus  validity  of  the  data. 

Recommendation.  Noted. 

4.1.6  Expand  Independent  Variables 

Observation.  Consider  expanding  the  study  to  include  significant  political  or 
economic  events  as  independent  variables  to  affect  refugee  numbers.  Significant  political 
changes  or  economic  sanctions  could  possibly  be  mapped  to  refugee  increases. 

Discussion.  Agree.  Political  violence  events  were  in  one  of  the  regression  models. 

Recommendation.  Noted  for  future  work. 

4.1.7  Lag 

Observation.  The  study  showed  that  a  24  week  lag  produced  the  best  correlation 
(using  regression  analysis  and  testing  different  lag  times  in  weeks)  between  violent 
events  and  refugees  (i.e.,  that  it  took  24  weeks  for  refugees  to  pack  up  and  go  to  a  refugee 
camp  after  a  series  of  violent  events.)  When  using  moving  averages  it  is  recommended 
that  the  number  of  weeks  be  varied  from  k  =  4  to  12. 

Discussion.  This  is  an  example  of  the  consequence  of  having  more  than  one 
method.  Lag  is  a  simple  regression  approach,  but  the  forecasting  models  lime  moving 
averages  were  better. 

Recommendation.  Sustain  testing  of  multiple  methods  where  possible  and 
continue  to  make  it  clear  that  the  weaker  method  is  not  recommended. 

4.1.8  Level  and  Duration  of  Violence 

Observation.  The  study  showed  that  a  24  week  lag  produced  the  best  correlation 
(using  regression  analysis  and  testing  different  lag  times  in  weeks)  between  violent 
events  and  refugees  (i.e.,  that  it  took  24  weeks  for  refugees  to  pack  up  and  go  to  a  refugee 
camp  after  a  series  of  violent  events.)  Is  the  level  and  duration  of  violence  or  both  also 
potentially  important? 
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Discussion.  There  were  data  limitations.  It  may  be  possible  to  use  dummy 
variable  for  duration.  Again,  as  in  4.1.7  above,  accuracy  of  the  results  showed  the 
forecasting  method  is  better. 

Recommendation.  Noted. 

4.1.9  Refuge  Flow 

Observation.  The  report  stated  that  the  team  desired  to  forecast  how  many 
refugees  would  enter  specific  refugee  camps  given  violence  in  Syria,  but  the  nature  of  the 
data  was  such  that  they  could  not  track  where  refugees  were  coming  from  so  it  was  hard 
to  project  where  they  would  end  up.  It  might  be  possible  to  forecast  where  refugees  might 
go  if  they  isolate  the  dependent  variable  to  just  one  of  the  refugee  camps  and  then  attempt 
to  do  the  same  multivariable  time  series  analysis  they  did  earlier  but  categorize  the 
violence  in  Syria  by  region.  The  analysis  may  show  the  violence  in  a  combination  of 
regions  may  be  highly  correlated  to  increases  in  refugees  in  a  specific  camp  while  others 
are  not.  If  feasible,  this  could  be  done  for  each  refugee  camp  in  turn. 

Discussion.  This  was  planned  for  the  third  phase  which  was  cancelled  given  the 
reduction  of  resources  for  this  project.  Consider  border  crossings  as  the  dependent 
variable;  however,  one  border  crossing  site  was  for  injured  only.  See  final  briefing. 

Recommendation.  Noted  for  future  work. 

4.1.10  Data  Cleaning 

Observation.  Much  of  data  analysis  involves  ‘cleaning  of  the  data.'  During  this 
step,  abnormalities  such  as  nonlinear  relationships,  nonconstant  variance,  discrepant 
observations,  outliers  (high  influencers),  and  dependence  are  observed  and  corrected  for. 
How  are  these  and  other  issues  with  the  data  mitigated  systematically? 

Discussion.  May  not  be  appropriate  for  the  deployed  analyst  in  this  situation  as  it 
may  be  beyond  their  training  level.  Intentionally  omitted  this  aspect  other  than  notes  on 
interpolated  data  and  notes  on  entering  dates. 

Recommendation.  Noted  for  appropriate  situations. 
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4.1.11  List  of  References 


Observation.  "The  most  common  means  for  measuring  the  accuracy  of  forecasting 
models  is  either  the  MAD  or  the  standard  deviation  of  the  forecasting  errors."  ~ 
Forecasting  User’s  Guide.  A  list  of  references  would  be  nice. 

Discussion.  A  list  was  developed,  but  not  provided. 

Recommendation.  Sustain. 

4.1.12  Test  Set 

Observation.  The  instructions  are  clear  about  how  to  manipulate  the  worksheet. 
There  does  not  seem  to  be  a  test  set  approach  involved.  Consider  this  may  be  a  potential 
technical  limitation. 

Discussion.  Considered,  but  not  appropriate  for  Excel. 

Recommendation.  Noted. 

4.1.13  Over-Fitting  the  Model 

Observation.  The  instructions  clearly  explain  how  to  manipulate  the  worksheet. 
Without  knowledge  of  what  the  settings  do,  however,  the  skilled  common  user  will  over- 
fit  of  the  model. 

Discussion.  Considered,  but  this  is  a  limitation  in  working  with  the  deployed 
analysts  with  limited  training  in  statistics.  Over- fitting  is  clearly  an  important  concern. 

Recommendation.  Noted  for  appropriate  situations. 

4.1.14  Time-Series  Data 

Observation.  It  seems  the  assumption  that  the  data  is  time-series  is  made,  but  not 

stated. 

Discussion.  This  technical  assumption  seems  obvious. 

Recommendation.  Noted. 
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4.1.15  Packaging  Documentation 

Observation.  The  document  titled,  'SecondDeliverable.doc'  seems  to  be  the 
primary  document.  No  document  in  the  .zip  file  provided  is  clearly  the  main  document. 

Discussion.  This  is  an  artifact  of  packaging  documents  for  each  phase.  There  is  a 
document  that  provides  context  for  both  deliverables  and  their  content. 

Recommendation.  Improve. 

4.1.16  DCMP  Worksheet 

Observation.  Overall  the  products  are  good.  The  DCMP  worksheet  has  a  simple 
methodology  that  is  beneficial  in  helping  the  staff  detennine  which  sources  are  of  the 
highest  quality  for  them  to  collect  data  for  analysis. 

Discussion.  Noted. 

Recommendation.  Sustain. 

4.1.17  Training  Sample 

Observation.  Overall  the  products  are  good.  The  regression  and  forecasting 
spreadsheets  achieve  their  objectives  for  the  information  that  is  important  for  the  decision 
makers;  however,  if  these  spreadsheets  are  being  used  by  non-ORSA's,  then  recommend 
some  type  of  training  exercise  be  included.  The  instruction  sheet  could  serve  as  a  back-up 
reference.  I  think  including  the  model  and  tricks  of  the  trade  included  in  the  instructions 
are  good  at  helping  non-ORSA's  understand  the  limitations  of  the  analysis. 

Discussion.  This  training  was  accomplished  at  WSMR  for  the  next  two  deploying 
analysts.  The  users  are  not  necessarily  non-ORSA’s;  however,  they  are  working  in  a 
deployed  setting. 

Recommendation.  Sustain. 

4.2  FOLLOW-UP  QUESTIONS 

Did  CJ-F  provide  requested  feedback  on  the  assessment  framework?  No  feedback 
due  to  priorities  of  deployed  analysts. 
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Has  CJ-F  reported  on  use  of  the  assessment  framework?  No  related  analysis,  but 
some  requests  for  briefing  products.  This  work  will  be  presented  at  AORS. 

Has  TRAC  or  others  conducted  any  subsequent  related  analysis?  No. 

Does  the  analysis  team  have  observations  to  add?  No. 

Was  an  AAR  conducted  for  this  analysis?  No  AAR  was  conducted  since  this  was 
a  one-analyst  effort  at  WSMR  supporting  the  deployed  analysts. 
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5.  CONCLUSION 


5.1  HUMANITARIAN  ASSISTANCE  IN  SYRIA 

TRAC  provided  an  assessment  framework  to  CF-J  that  included  measures  and 
data  sources  to  identify  the  basic  needs,  match  those  needs  with  appropriate  type  and 
quantity  of  HA,  and  assess  effectiveness  after  HA  has  been  delivered  to  refine 
coordination.  TRAC  successfully  delivered  two  sets  of  products  to  CF-J  to  meet  this 
need.  The  technical  review  produced  17  specific  observations.  No  substantial  weakness 
was  found  in  the  technical  products  produced  for  this  project  and  several  areas  for 
sustaining  the  technical  merit  of  the  analysis  were  identified. 

5.2  TRAC  TECHNICAL  REVIEW  PROGRAM  PROTYPING 

This  review  prototyped  the  emerging  TRAC  Technical  Review  Process  to  be 
implemented  in  FY15.  It  demonstrated  the  overall  feasibility  and  potential  value  of  the 
approach;  however,  a  few  practical  implementation  challenges  were  identified. 

The  most  serious  challenge  was  maintaining  the  anonymity  of  the  contributors. 
The  internet  based  method  for  submitting  comments  was  insufficient  and  in  several  cases 
it  was  difficult  to  determine  what  aspect  of  the  analysis  was  referenced  in  the  comment. 
Providing  more  detailed  and  specific  guidance  to  contributors  to  document  comments 
will  reduce  this  challenge  and  the  other  challenges  identified.  A  method  to  dialogue  about 
the  comments  (e.g.,  anonymous  discussion  board)  should  eliminate  the  challenge. 
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